Added ProgressThrottler#70
Conversation
Added unit tests and integration tests for ProgressThrottler
juaninf
left a comment
juaninf
left a comment
There was a problem hiding this comment.
Seems ProgressThrottler is added as a standalone utility class but it is not integrated into BulkScanWorker, which seems is the natural way where it should be. BulkScanWorker already has the progressConsumer pipeline in handle(). Putting it with the throttler here would make throttled partial result persistence a built-in framework feature rather than something each subclass has to wire up manually.
|
ProgressThrottler is now integrated into BulkScanWorker.handle(), which wraps the progressConsumer with throttling before passing it to scan(). |
Throttled partial result publishing is now a built-in framework feature BulkScanWorker.handle() wraps the progressConsumer transparently. Config (partialResultsEnabled, partialResultThrottleMs) added to ScanConfig.
|
So - maybe we discuss tomorrow - but I dont think this PR makes sense. The crawler just fires events when things happen, these events trigger the callback - if the callback then needs to write to a database, log to a file - or perform a small cmd animation - is up to the callback. So - if a user wants to do throtteling - they should do it within their callback no? |
3 participants
Added ProgressThrottler, a generic throttler for progress updates during scans. Ensures updates are published at most once per configurable time window